System and method for lung visualization using ultasound

ABSTRACT

A system for ultrasound interrogation of a lung includes a memory, an electromagnetic (EM) board, an extended working channel (EWC), an EM sensor, a US transducer, and a processor. The memory stores a three dimensional (3D) model, a pathway plan for navigating a luminal network. An EM board generates an EM field. The EWC is configured to navigate the luminal network of a patient toward a target following the pathway plan and the EM sensor extends distally from the EWC and senses the EM field. The US transducer extends distally from a distal end of the EWC and generates US waves and receives US waves reflected from the luminal network and the processor processes the sensed EM field to synchronize a location of the EM sensor in the 3D model, to process the reflected US waves to generate images, or to integrate the generated images with the 3D model.

BACKGROUND

1. Technical Field

The present disclosure relates to systems and methods for visualizing alung using ultrasound imaging techniques. More particularly, the presentdisclosure relates to systems and methods that augment images of aluminal networks obtained by other imaging modality with ultrasoundimages.

2. Discussion of Related Art

Standard of care for lung diseases, such as asthma, chronic obstructivepulmonary disease (COPD), and chronic obstructive lung disease (COLD),or for lung-related diseases has been focused largely on medical and/ordrug management which are highly invasive to patients in general. Forexample, it has been reported for decades that lung denervation vialocalized and invasive means (e.g., surgery) may provide therapeuticbenefit for asthma or emphysema.

Electromagnetic navigation (EMN) has helped expand the possibilities oftreatment of luminal networks such as the lungs. EMN relies onnon-invasive imaging technologies, such as computed tomography (CT)scanning, magnetic resonance imaging (MRI), or fluoroscopictechnologies. EMN in combination with these non-invasive imagingtechnologies has been also used to identify a location of a target andto help clinicians navigate a luminal network of the lung to the target.However, images generated by these non-invasive imaging technologieshave been unable to provide a resolution sufficient to identify featuressuch locations of nerves that run parallel to the luminal network.Further, when a treatment is performed, additional images using thesenon-invasive imaging technologies must have been performed to determinewhether the treatment has been complete. That increases the number ofexposures of harmful X-rays or substances to the patient and costs oftreatments. Still further, every clinician is desirous of a greaterresolution of the area being treated. Accordingly there is a need for animaging modality, which provides the desired resolution and isclinically efficient in operation.

SUMMARY

In an aspect, the present disclosure features a system for US basedinterrogation of a lung. The system includes a memory, anelectromagnetic (EM) board, an extended working channel (EWC), an EMsensor, a US transducer, and a processor. The memory stores a threedimensional (3D) model of a luminal network and a pathway plan fornavigating a luminal network and the EM board is configured to generatean EM field. The EWC is configured to navigate the luminal network of apatient toward a target in accordance with the pathway plan and the EMsensor extends distally from a distal end of the EWC and is configuredto sense the EM field. The US transducer is configured to generate USwaves and receive US waves reflected from the luminal network and theprocessor is configured to process the sensed EM field to synchronize alocation of the EM sensor in the 3D model, to process the reflected USwaves to generate US images, or to integrate the generated images withthe 3D model.

In another aspect, the system further includes a display deviceconfigured to display the integrated 3D model and US images. The displayis further configured to display a status based on the location of theEM sensor. The status indicates whether the EM sensor is located at anot-in-target location, the target, or a location adjacent to healthytissue. The status further indicates whether treatment of the target iscomplete.

In another aspect, a resolution of the generated images is finer than aresolution of the 3D model.

In another aspect, the EM sensor is located at or around a distal end ofthe EWC.

In another aspect, the system further includes a plurality of referencesensors located on a patient and configured to create a breathingpattern of the patient. The system still further includes a trackingdevice, which is coupled to the plurality of reference sensors and theEM sensor, and is configured to identify the location of the EM sensorby compensating for patient's breathing based on the breathing pattern.

In another aspect, a location of integration of the generated images isbased on the location of the EM sensor in the 3D model.

In another aspect, the processor is further configured to identifytissue density based on the reflected US waves. The processor is stillfurther configured to determine whether a treatment device is at acenter of the target.

In yet another aspect, the processor is further configured to determinea sufficiency of treatment based on a density of the target according tothe reflected US waves.

In yet another aspect, the processor is further configured to detect asize of the target.

In yet another aspect, the processor is further configured to determineshrinkage of the target real-time during and after a treatment of thetarget.

In another aspect, the generated images show outside of the luminalnetwork.

In another aspect, the processor is further configured to determine anoffset between the EM sensor and the US transducer. Integration of thegenerated images with the 3D model is based on the offset.

In yet another aspect, the US transducer is positioned in a forwardlooking manner before the EM sensor.

Any of the above aspects and embodiments of the present disclosure maybe combined without departing from the scope of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Objects and features of the presently disclosed systems and methods willbecome apparent to those of ordinary skill in the art when descriptionsof various embodiments are read with reference to the accompanyingdrawings, of which:

FIG. 1 is a perspective view of a system for visualizing a lung of apatient in accordance with an embodiment of the present disclosure;

FIG. 2A is a profile view of a catheter guide assembly in accordancewith an embodiment of the present disclosure;

FIG. 2B is an expanded view of the indicated area of detail, which showsa distal tip of an extended working channel of FIG. 2A in accordancewith an embodiment of the present disclosure;

FIG. 3 is an anatomical illustration of a three dimensional model for alung in accordance with an embodiment of the present disclosure;

FIG. 4A is an illustration of a pathway from the entry point to thetarget in accordance with an embodiment of the present disclosure;

FIG. 4B is a transverse cross-sectional view of the section of the lungof FIG. 4A taken along section line B-B;

FIG. 4C is an illustration of a catheter guide assembly inserted into alung following the pathway plan of FIG. 4A;

FIG. 4D is an enlarged detail view of the circled area of FIG. 4C;

FIG. 5A is a flowchart of a method for visualizing a lung using US wavesin accordance with an embodiment of the present disclosure;

FIG. 5B is a flowchart of a method for navigation to the target inaccordance with an embodiment of the present disclosure; and

FIG. 5C is a flowchart of a method for checking the level of treatmentin accordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure is related to systems and methods for visualizingthe luminal network of a lung using ultrasound (US) imaging technologieswhich provide a sufficient resolution to identify and locate a targetfor diagnostic, navigation, and treatment purposes. US imaging,particularly in conjunction with non-invasive imaging can provide agreater resolution and enable luminal network mapping and targetidentification. Further, additional clarity is provided with respect totissue adjacent identified targets which can result in differenttreatment options being considered to avoid adversely affecting theadjacent tissue. Still further, the use of US imaging in conjunctionwith treatment can provide detailed imaging for post treatment analysisand identification of sufficiency of treatment. Although the presentdisclosure will be described in terms of specific illustrativeembodiments, it will be readily apparent to those skilled in this artthat various modifications, rearrangements, and substitutions may bemade without departing from the spirit of the present disclosure. Thescope of the present disclosure is defined by the claims appended tothis disclosure.

FIG. 1 illustrates an electromagnetic navigation (ENM) system 100, whichis configured to augment CT, MRI, or fluoroscopic images, with US imagedata assisting in navigation through a luminal network of a patient'slung to a target. One such ENM system may be the ELECTROMAGNETICNAVIGATION BRONCHOSCOPY® system currently sold by Covidien LP. Thesystem 100 includes a catheter guide assembly 110, a bronchoscope 115, acomputing device 120, a monitoring device 130, an EM board 140, atracking device 160, and reference sensors 170. The bronchoscope 115 isoperatively coupled to the computing device 120 and the monitoringdevice 130 via wired connection (as shown in FIG. 1) or wirelessconnection (not shown).

The bronchoscope 115 is inserted into the mouth of the patient 150 andcaptures images of the luminal network of the lung. In the EMN system100, inserted into the bronchoscope 115 is a catheter guide assembly 110for achieving access to the periphery of the luminal network of thepatient 150. The catheter guide assembly 110 may include an extendedworking channel (EWC) 230 into which a locatable guide catheter (LG) 220with EM sensor 265 (FIG. 2B) at the distal tip is inserted. EWC 230, theLG 220, and an EM sensor 265 are used to navigate through the luminalnetwork of the lung as described in greater detail below.

The computing device 120, such as, a laptop, desktop, tablet, or othersimilar computing device, includes a display 122, one or more processors124, memory 126, a network card 128, and an input device 129. The system100 may also include multiple computing devices, wherein the multiplecomputing devices 120 are employed for planning, treatment,visualization, or helping clinicians in a manner suitable for medicaloperations. The display 122 may be touch-sensitive and/orvoice-activated, enabling the display 122 to serve as both an input andoutput device. The display 122 may display a two dimensional (2D) imagesor three dimensional (3D) model of a lung to locate and identify aportion of the lung that displays symptoms of lung diseases. Thegeneration of such images and models is described in greater detailbelow. The display 122 may further display options to select, add, andremove a target to be treated and settable items for the visualizationof the lung. In an aspect, the display 122 may also display the locationof the catheter guide assembly 110 in the luminal network of the lungbased on the 2D images or 3D model of the lung. For ease of descriptionnot intended to be limiting on the scope of this disclosure, a 3D modelis described in detail below but one of skill in the art will recognizethat similar features and tasks can be accomplished with 2D models andimages.

The one or more processors 124 execute computer-executable instructions.The processors 124 may perform image-processing functions so that the 3Dmodel of the lung can be displayed on the display 122. In embodiments,the computing device 120 may further include a separate graphicaccelerator (not shown) that performs only the image-processingfunctions so that the one or more processors 124 may be available forother programs.

The memory 126 stores data and programs. For example, data may be imagedata for the 3D model or any other related data such as patients'medical records, prescriptions and/or history of the patient's diseases.One type of programs stored in the memory 126 is a 3D model and pathwayplanning software module (planning software). An example of the 3D modelgeneration and pathway planning software may be the ILOGIC® planningsuite currently sold by Covidien LP. When image data of a patient, whichis typically in digital imaging and communications in medicine (DICOM)format, from for example a CT image data set (or image data set by otherimaging modality) is imported into the planning software, a 3D model ofthe bronchial tree is generated. In an aspect, imaging may be done by CTimaging, magnetic resonance imaging (MRI), functional MRI, X-ray, and/orany other imaging modalities. To generate the 3D model, the planningsoftware employs segmentation, surface rendering, and/or volumerendering. The planning software then allows for the 3D model to besliced or manipulated into a number of different views including axial,coronal, and sagittal views that are commonly used to review theoriginal image data. These different views allow the user to review allof the image data and identify potential targets in the images.

Once a target is identified, the software enters into a pathway planningmodule. The pathway planning module develops a pathway plan to achieveaccess to the targets and the pathway plan pin-points the location andidentifies the coordinates of the target such that they can be arrivedat using the EMN system 100, and particularly the catheter guideassembly 110 together with the EWC 230 and the LG 220. The pathwayplanning module guides a clinician through a series of steps to developa pathway plan for export and later use in during navigation to thetarget in the patient 150. The term, clinician, may include doctor,surgeon, nurse, medical assistant, or any user of the pathway planningmodule involved in planning, performing, monitoring and/or supervising amedical procedure.

Details of these processes and the pathway planning module can be foundin concurrently filed with this disclosure and commonly assigned U.S.Patent Application No. 62/035,863 filed Aug. 11, 2014 entitled“Treatment procedure planning system and method” and U.S. patentapplication Ser. No. 13/838,805 filed by Covidien LP on Jun. 21, 2013,and entitled “Pathway planning system and method,” the entire contentsof each of which are incorporated in this disclosure by reference. Suchpathway planning modules permit clinicians to view individual slices ofthe CT image data set and to identify one or more targets. These targetsmay be, for example, lesions or the location of a nerve which affectsthe actions of tissue where lung disease has rendered the lung functioncompromised.

The memory 126 may store navigation and procedure software whichinterfaces with the EMN system 100 to provide guidance to the clinicianand provide a representation of the planned pathway on the 3D model and2D images derived from the 3D model. An example of such navigationsoftware may be the ILOGIC® navigation and procedure suite sold byCovidien LP. In practice, the location of the patient 150 in the EMfield generated by the EM field generating device 145 must be registeredto the 3D model and the 2D images derived from the model.

Such registration may be manual or automatic and is described in detailin concurrently filed with this disclosure and commonly assigned U.S.Patent Application 62/020,240 filed by Covidien LP on Jul. 2, 2014, andentitled “System and method for navigating within the lung.”

As shown in FIG. 1, the EM board 140 is configured to provide a flatsurface for the patient to lie down and includes an EM field generatingdevice 145. When the patient 150 lies down on the EM board 140, the EMfield generating device 145 generates an EM field sufficient to surrounda portion of the patient 150. The EM sensor 265 at the distal tip 260 ofthe LG 220 is used to determine the location of the EM sensor 265 in theEM field generated by the EM field generating device 145.

In embodiment, the EM board 140 may be configured to be operativelycoupled with the reference sensors 170 which are located on the chest ofthe patient 170. The reference sensors 170 move up and down followingthe chest while the patient 150 is inhaling and move down following thechest while the patient 150 is exhaling. The movement of the referencesensors 170 in the EM field is captured by the reference sensors 170 andtransmitted to the tracking device 160 so that the breathing pattern ofthe patient 150 may be recognized. The tracking device 160 also receivesoutputs of the EM sensor 265, combines both outputs, and compensates thebreathing pattern for the location of the EM sensor 265. In this way,the location identified by the EM sensor 265 may be compensated for sothat the compensated location of the EM sensor 265 is synchronized withthe 3D model of the lung. Once the patient 150 is registered to the 3Dmodel, the position of the EWC 230 and particularly the LG 220 can betracked within the EM field generated by the EM field generator 145, andthe position of the LG 220 can be depicted in the 3D model or 2D imagesof the navigation and procedure software.

FIG. 2A illustrates an embodiment of the catheter guide assembly 110 ofFIG. 1. The catheter guide assembly 110 includes a control handle 210.The control handle 210 has an actuator and a selector mechanism forselectively mechanically steering, rotating, and advancing an extendedworking channel (EWC) 230 or locatable guide catheter (LG) 220 insertedin the EWC 230, meaning that the distal tip 260 of the LG 220 is turningto a direction in accordance with the movement of the control handle210. A locking mechanism 225 secures the EWC 230 and the LG 220 to oneanother. Catheter guide assemblies usable with the instant disclosureare currently marketed and sold by Covidien LP under the nameSUPERDIMENSION® Procedure Kits and EDGE™ Procedure Kits. For a moredetailed description of the catheter guide assemblies is made tocommonly-owned U.S. patent application Ser. No. 13/836,203 filed on Mar.15, 2013 by Ladtkow et al. and U.S. Pat. No. 7,233,820, the entirecontents of which are hereby incorporated by reference.

FIG. 2B is an expanded view of the distal end 250 of the EWC 230 of FIG.2A. A US transducer 265 located at the distal end 250 of the EWC 230.The EM sensor 265 is located at the distal tip 260 of the LG 220, whichis depicted extending beyond the distal end 250 of the EWC 230. Asdescribed briefly above, the EM sensor 265 senses the EM field generatedby the EM field generating device 145. The sensed EM field is used toidentify the location of the EM sensor 265 in accordance with thecoordinate system of the EM field. When the location of the EM sensor265 is determined by the tracking device 160, the computing device 120compares the location of the EM sensor 265 with the 3D model of the lungand registers the location of the EM sensor 265 into the coordinatesystem of the 3D model.

For example, when the EM sensor 265 is near at the entrance to thetrachea, the EM sensor 265 senses the EM field and the location of theEM sensor is then compared with the trachea portion of the 3D model sothat the location of the EM sensor 265 is depicted in the correspondinglocation of the 3D model and 2D images of the navigation and proceduresoftware. And when the EM sensor 265 is further inserted through thetrachea to a location where separate bronchial trees are branched, thedistance the EM sensor 265 travels from the entrance of the trachea tothe branching location is scaled to match to the corresponding distancein the 3D model and 2D images of the navigation and procedure software.Specifically, when the EM sensor 265 travels along the trachea, thedistance is measured in accordance with the coordinate system of the EMfield. Since the coordinate system of the EM field is different from thecoordinate system of the 3D model, there is a scaling factor to matchthe coordinate system of the EM field to the coordinate system of the 3Dmodel. Thus, by multiplying a scale factor to the distance the EM sensor265 travels, the coordinate system of the EM field is synchronized withthe coordinate system of the 3D model. In this way, the EM field may besynchronized with the 3D model and 2D images of the navigation andprocedure software. Or other suitable method may be employed tosynchronize the coordinate system of the EM field with the coordinatesystem of the 3D model.

As noted above, the 3D model may not provide a resolution sufficient forguiding the EWC 230 of the catheter guide assembly 110 to a target,meaning that the 3D model becomes blurred or ceases to recognize theluminal network as the EWC 230 approaches a certain point. For example,when CT scan images are taken by 1 mm thick and 1 cm apart by a CT scandevice, corresponding 3D model and/or pathway plans may not be able toshow full perspective of a target whose size is less than 1 cm or aportion of a luminal network whose diameter is less than 1 cm. Thus,another imaging modality is necessary to find and/or identify a targetand/or a terminal bronchial branch, whose size is less than a certainsize which CT scan images are unable to show with sufficient details.For this purpose, the memory 126 also stores another program that canprocess and convert image data captured by an imaging modalityassociated with the catheter guide assembly 110, as will be described indetail below. This image data may be converted into visual images havingsufficient resolutions to identify such targets and terminal bronchialbranches or be incorporated into and used to update the data from the CTscans in an effort to provide a greater resolution and fill-in data thatwas missing in the CT scan.

One such imaging modality is depicted in FIG. 2B where the US transducer255 is depicted on the EWC 230 proximal the distal end. One of skill inthe art will recognize that the location of the US transducer 255 andthe EM sensor 265 may be alternated between the LG 220 and the EWC 230,or that more than one of each sensor and transducer may be employedwithout departing from the scope of the present disclosure. The UStransducer 255 transmits ultrasound waves and receives reflectedultrasound waves. Generally, ultrasound waves penetrate tissue based onthe frequency of the ultrasound waves. For example, 1 megahertz (MHz)ultrasound waves penetrate to a depth of 2 cm to 5 cm and 3 MHzultrasound waves penetrate to a depth of 1.5 cm. Thus, US waves aresuitable for imaging bronchial trees. In an aspect, the US transducer255 may be a radial US transducer.

Generally, US waves are reflected at a boundary where density changes orat the interface between tissues. While the US transducer 255 isnavigating the luminal network of the lung, the US waves are reflectedfrom the inside wall of a bronchial tree, from the outside wall of thebronchial tree, and from a diseased portion or cancerous portion locatedat the outside wall of the bronchial tree and provide finite details ofthe lung structure and the tissue patency that could not otherwise berevealed using non-invasive imaging means.

The reflected US waves have information such as amplitude and a delayedtime between transmission of the US waves and reception of the reflectedUS waves. Since the US waves travels differently and attenuatesamplitudes differently in accordance with the density of tissue, theamplitude and the delayed time may be used to identify a type of tissue,a density of the tissue, and/or a size of the tissue. Since the densityof abnormal tissues (e.g., diseased or cancerous cells) are differentfrom the normal lung tissue, the reflected US waves may be used toidentify the diseased or cancerous cells from normal cells and the sizeand/or thickness of the diseased or cancerous cells.

The computing device 120 analyzes the reflected US waves and generatesvisual images which has a higher resolution than that of the 3D model orthe CT scan images. The generated visual images may be augmented to andintegrated with the 3D model of the lung or 2D images such as the CTscan images.

In embodiments, when a treatment is performed to treat an abnormaltissue located at the outside wall of a bronchial tree, generally, thesize of the abnormal tissue shrinks and density of the abnormal tissuechanges to the density of the normal lung tissue. Traditionally, when atreatment is performed, another CT scan is performed to obtain anotherset of CT images to check the size of the diseased or cancerous cells sothat clinicians may determine whether the treatment is complete oranother one is to be made. Since the US transducer 255 is able to checkthe size and the density of the abnormal tissue, the level of treatmentmay also be checked at the spot without performing another CT scan.

As shown in FIG. 2B, the US transducer 255 and the EM sensor 265 areseparated by a distance, D_(OFF). This distance, D_(OFF), may be sensed,coded into the navigation and procedure software, measured and sent bythe clinician, or sensed by the US transducer 255 and the EM sensor 265.The computing device 120 uses the distance, D_(OFF), to adjust theincorporation of the US images into the 3D model or 2D images derivedtherefrom. For example, when the EM sensor 265 is located at the distaltip 260 of the LG 220, the US transducer 255 is located at orcircumscribing the distal end 250 of the EWC 230, and both sensors are 1cm distance apart from each other, this distance is recognized by thesoftware and the US data or images is offset and integrated into the 3Dmodel or 2D images derived therefrom by a distance in the coordinatesystem of the 3D model, which corresponds to 1 cm in the coordinatesystem of the EM field.

When the EWC 230 and the LG 220 reaches a target by manipulation of thecatheter guide assembly 110 following the pathway plan, the EM sensor265 confirms its location at the target and a clinician may visuallyconfirm the location at the target by looking at visual images generatedfrom the US images. The LG catheter 220 may be removed from the catheterguide assembly 110 and a biopsy tool may be inserted into the EWC 230 tothe target to retrieve sample of the target for confirmation of thedisease. An anchoring tool may be employed to anchor the EWC 230 at thetarget. Further, treatment tools such as an ablation catheter may beinserted through the EWC 230 and into the target. The US transducer 255may then be used to transmit and receive US waves and the computingdevice 120 determines whether the treatment tool is at the epicenter ofthe target by comparing the densities of the tissue surrounding thetreatment tool or by generating US images of the target for clinicalcomparison. By being located at the epicenter of the target, thetreatment tool may perform treatment with high efficiency. In an aspect,the EM sensor 265 and the US transducer 255 may be located at or aroundthe EWC 230 with a distance apart from each other or at or around the LG220 with a distance apart from each other.

In embodiments, the US transducer 255 and the computing device 120 maycheck the size of the target either before or after treatment. When thesize of the target is greater than a threshold size, another treatmentmay be necessary to complete the treatment. Thus, the treatmentcontinues until the size of the target is decreased under the thresholdsize. In this way, visualization using US waves may be utilized forchecking the level of treatment.

In embodiments, the US transducer 255 may be a sacrificial US transducer255 which may be positioned in a forward looking manner to identify thetarget. The US transducer 255 is sacrificial because it may be renderedineffective following treatments of the target by the application ofmicrowave energy of the treatment device.

In embodiments, in a pre-treatment step, one or more markers can beplaced through the EWC 230 to identify the location of the target. Themarker may assist in navigating to a desired location and confirmingplacement of the EWC 230, particularly after removal of the LG 220 andthe EM sensor 265 when the EM navigation features of the presentdisclosure may not be effective. The marker may give a clinician anability to re-visit the target after the target has been treated and tocollect further samples. The marker may be a fiducial marker,fluorescent dye, or FLUOROGOLD®. In the case of fluorescent dye markers,the US imaging capabilities may further increase the determination ofsufficiency of treatment, or provide greater clarity as to the exactlocation of the target. Other markers for marking the location of atarget may be employed by those of ordinary skill in the art withoutdeparting from the scope of the present disclosure.

FIG. 3 illustrates a 3D model 300 for a patent's bronchial trees and thetrachea together with the lung. The 3D model 300 may include informationof most of the organs so that a clinician may selectively see particularorgans or portions of organs of interest as shown in FIG. 3. In thiscase, these selected organs are the lungs including right lobe 310, theleft lobe 320, the trachea 330 and bronchial trees 340. The right lobe310 has three sub-lobes, i.e., superior lobe 312, middle lobe 314, andinferior lobe 316, and the left lobe 320 has two sub-lobes, i.e.,superior lobe 322 and inferior lobe 324.

The trachea 330 is a tube that connects the pharynx and larynx to thelung 310 and 320. At the lower end of the trachea 330, left or rightprimary bronchus 342 is divided. Secondary bronchus 344 also divides atthe lower end of the primary bronchus 342. The circumference of theprimary bronchus 342 is greater than that of the secondary bronchus 344.In the same manner, tertiary bronchus 346 divides at the lower end ofthe secondary bronchus 344 and terminal bronchiole 348 divides at thelower end of the tertiary bronchus 346. The primary bronchus 342, thesecondary bronchus 344, and the tertiary bronchus 346 are supported bycartilaginous plates. However, when the size of the tertiary bronchus346 becomes smaller and smaller, the cartilaginous plates disappear andouter wall is dominated by smooth muscle. The outer wall of the terminalbronchiole 348 is also dominated by smooth muscle.

Diseased or cancerous cells or simply a target may exist on anybronchial trees, the primary bronchus 342, the secondary bronchus 344,the tertiary bronchus 346, and the terminal bronchioles 348. No matterwhere a target is located, when a target is too small to be detected bya CT imaging modality, the target may still be detected by the USimaging modality while the EWC 230 with US transducer 255 is navigatingtoward another target through the luminal network of the lung. The UStransducer 255 provides greater specificity and greater accuracy indetecting and identifying a target's location in the patient. Inaccordance with at least one embodiment, the US transducer 255 may be aradial ultrasound transducer employed to further refine the image dataof the lungs by following the pathway plan described above and capturingUS image data along the pathway. This US image data may be registered tothe CT scan images and/or the 3D model 300 to provide greater claritywith respect to the detection, location, and size of a target. Forexample, this data may also be used diagnostically to help the clinicianconfirm that all likely targets have been identified or treatedcompletely after treatments.

In addition, when the US transducer 255 captures image data the capturedimage data is transferred to the computing device 120 wirelessly or viaa wired connection. Image data captured by an ultrasound imagingmodality, is not yet readily apprehended by a clinician. The computingdevice 120 processes and converts it to an image with which a cliniciancan identify a type of tissue, diagnose a disease, identify a locationof the catheter guide assembly 110, which is the place of image taking,or determine a level of treatment.

FIG. 4A shows a planar view of bronchial trees of the 3D model or of theslices of images of the lung such as the bronchial trees of FIG. 3 and apathway plan to a target. When a target is located at the tip of thebottom left end of the terminal bronchiole of FIG. 3, a pathway planshows how to get to the target via the luminal network of the lung.

FIG. 4B shows an expanded transverse cross-sectional view of theterminal bronchiole of FIG. 4A taken along section line B-B. Theterminal bronchiole is surrounded by smooth muscle 405. Nerves 410 andveins 415 are located on the outer wall of the smooth muscle 405. The USimaging modality, as described above, provides a local view of theairways even out to the terminal bronchiole so that even the thin nerves410 and the veins 415 on the smooth muscle 405 can be detected andidentified. Thus, by using US imaging in addition to the CT imaging,navigation to and direction of therapies such as denervation can beaccomplished even at the lung periphery enabling greater granularity oftreatment options and with greater precision.

FIG. 4C illustrates a bronchoscope 420 with a catheter guide assemblyinserted into the lungs via a natural orifice (e.g., the mouth) of apatient toward the target following a pathway plan. When thebronchoscope 420 reaches a certain location of the lung, thebronchoscope 420 becomes wedged and cannot go further into bronchialtree due to the size constraints. Then, the EWC 430 of the catheterguide assembly may be used to navigate the luminal network to a target450 following the pathway plan, as described above. The EWC 430 is smalland thin enough to reach the target 450. FIG. 4D illustrates an enlargeddetail view of the circled area of FIG. 4C, where a locatable guide (LG)may stick out of the distal tip of the EWC 430 which navigates theluminal network to the target 450 located at the terminal bronchiole ofthe lung.

FIG. 5A is a flowchart of a method 500 for visualizing a lung using USimaging technology. The method 500 starts at step 505 by importing a 3Dmodel of a lung and a pathway plan to a target into the navigation andprocedure software stored on a computer such as the computing device 120of FIG. 1.

In step 510, an EM field is generated by an EM board, such as the EMfield generating device 145 of the EM board 140 as shown in FIG. 1. Instep 515, an EM sensor 265 and a US transducer 255 are inserted into thelung via a natural orifice or an incision. The EM sensor 265 and the UStransducer 255 may be located on the EWC 230 with a distance apart ormay be located at different places. For example, the EM sensor 265 maybe located at or around the distal tip 260 of the LG 220 and the UStransducer 255 may be located at or around the distal end 250 of the EWC230, or vice versa.

In step 520, the EM sensor 265 senses the EM field and the sensedresults are transmitted to the computing device 120. The sensed resultsare used to calculate a location of the EM sensor 265 in the coordinatesystem of the EM field. When the location is calculated, the computingdevice compares the location of the EM sensor 265 with the 3D model, the2D images derived therefrom, and the pathway plan. In an aspect, thelocation of the EM sensor 265 may be compensated according to thebreathing pattern of the patient by the tracking device 160 and thereference sensors 170 before transmitted to the computing device. Thus,the location of the ME sensor 255 may not vary in the coordinate systemof the 3D model while the patient inhales or exhales.

In step 525, the location of the EM sensor 265 is synchronized to the 3Dmodel and the 2D images derived therefrom. This location may be thestarting location of the 3D model, or the entrance of the trachea of the3D model. Even though the location is synchronized, the actual movementof the EM sensor 265 is not synchronized to the 3D model yet, here.

The EM sensor 265 travels a certain distance (e.g., from the entrance ofthe trachea to the branching point at the bottom of the trachea). Thisdistance may be measured in the coordinate system of the EM field afterthe EM sensor 265 starts to sense the EM field. In step 530, thetravelling distance by the EM sensor 265 according to the coordinatesystem of the EM field may be scaled so that the scaled distance ismatched to the coordinate system of the 3D model. After this step, thelocation and the movement of the EM sensor 265 are substantially mappedinto the 3D model. This is the synchronization or registration of thepatient to the 3D model and the 2D images derived therefrom.

In step 535, the EM sensor 265, the LG 220, and the EWC 230 navigate theluminal network of the lung to the target following the pathway plan. Instep 540, it is determined whether the sensor 265 has reached thetarget. If it is determined that the EM sensor 265 has not reach thetarget, step 535, i.e., the navigation step, is continued until thetarget is reached following the pathway plan

In embodiments, when it is determined that the target is reached in step540, step 545 may be performed to image the target with the UStransducer 255 to confirm its location. This may involve confirmingtissue densities or confirming position relative to markers and otherlocation confirmatory steps. In addition, imaging of the target may beemployed after treatment to ensure sufficiency of treatment. Step 545 isdescribed in further detail in FIG. 5C below.

FIG. 5B shows detail steps of navigation to the target, step 535 of themethod 500 of FIG. 5A. In step 550 US waves are transmitted by the UStransducer 255 while the distal end of the EWC 230 navigates to thetarget following the pathway plan. In step 555, the US transducer 255receives and sends US waves reflected from the lung tissue to thecomputing device 120, which in turn processes the reflected US waves instep 560. The reflected US waves have information such as amplitude anddelayed time from the transmission to the reception. The computingdevice 120 process the information to determine the density or size ofthe lung tissue and/or determine whether there are new targets (i.e.,diseased or cancerous cells to be treated) not found in the CT scanimages.

In step 565, it is determined whether there is a new target along thepathway plan to the target. When it is determined that there is a newtarget, in step 570, the new target is identified and registered to the3D model for later treatment. In step 575, the route to the new target,which is a part of the pathway plan to the target, is also saved as apathway plan to the new target. Then, the method 535 goes back to step565 to continue checking whether there are any further new targets.

When it is determined that there is no new target in step 565, thecomputing device may generate images based on the processed reflected USwaves. Since the US waves are reflected from an interface betweentissues where density changes, the generated images show details bothinside and outside of the bronchial tree. The generated images maydepict a diseased or cancerous cells residing on the outside of thebronchial tree. In an aspect, when a treatment device penetrates thetarget for treatment purposes, the generated images can also be used toshow whether the treatment device is in the center of the target.

In step 585, the generated images are integrated into the 3D model basedon the location of the EM sensor 265 and the offset distance D_(OFF)between the EM sensor 265 and the US transducer 255. In embodiments, thegenerated images may be overlaid on CT scan images so that a lowerresolution portion of the CT scan images may be replaced with a higherresolution images (i.e., the generated US images), the image data may beselectively fused to create a composite image data set, or the data canbe incorporated into the CT image data. In step 590, the computingdevice displays the generated images with the 3D model or simply theintegrated 3D model. These steps 550-590 of navigation are repeateduntil the target is reached as shown in the method 500 of FIG. 5A.

In an embodiment, visualization using the US waves may also be used todetermine the sufficiency of treatment. When one treatment is performedon a target, the attributes of the target including size, density, andwater content of the target is generally altered. Thus, in order tocheck whether the treatment is complete, the attributes of the targetmust be checked and compared to similar measurements taken beforetreatment. FIG. 5C illustrates a flowchart of a method for checking thesufficiency of treatment after it is determined that the EM sensor 265reaches the target in step 540 of FIG. 5A. In step 605, a treatmentdevice, such as an ablation catheter, is inserted into the EWC 230 afterremoval of the LG 220 and its EM sensor 265. In step 610, it isdetermined whether the treatment device is at the epicenter of thetarget. This is done by use of the US transducer 255. US images showwhere the density of imaged tissue changes and the target has adifferent density from normal lung tissue.

When it is determined that the treatment device is not at the epicenterof the target, the treatment device is inserted or retreated more orless to adjust its location in step 615. Then, in step 610, the locationof the treatment device is again checked. When it is determined that thetreatment device is located at the epicenter of the target in step 610,the treatment device treats the target.

In embodiments, similar steps as steps 605-615 of FIG. 5C may be appliedfor biopsy. When a biopsy tool is inserted to take samples of thetarget, the US transducer 255 is used to check whether the biopsy toolis at the correct location of the target. When it is determined that thebiopsy tool is at the right place, then the biopsy tool takes samples.Or when it is determined that the biopsy tools is not at the target, thebiopsy tool may be adjusted to reach correctly at the target.

In step 620, the treatment device treats the target. Following treatmentapplication, the US transducer 255 may be employed to image the target,determine the attributes of the target in step 625 (e.g., the size), andcompares the attributes of the target with threshold values in step 630.Here, the threshold size may be predetermined based on a type of diseaseand may indicate that the disease is treated completely.

When it is determined that the size of the treated target is greaterthan the threshold size, the computing device 120 notifies a clinicianof incomplete treatment by displaying on the display screen such noticein step 635. The method 545 then goes back to step 620 for anothertreatment. These steps 620-635 repeat until the treatment is complete.In an aspect, these treatments may be performed at the spot or for aperiod. In a case when the treatments are performed during a period, amarker may be placed at or near the target so that a treating device canbe inserted to the target with certainty during a later treatment.

When it is determined that the size of the target is less than or equalto the threshold size in step 630, the computing device 120 notifies aclinician of complete treatment by displaying that the treatment iscomplete in step 640, and the method 545 of checking the level oftreatment is ended. Thus, the US transducer 255 and US imaging featuresof the present disclosure may be employed to confirm the sufficiency oftreatment of a target.

In another embodiment, the monitoring device 130 and/or the computer 120may display a color code on the display, notifying a clinician of astatus. The status may be based on a location of the EWC 230 of thecatheter guide assembly 110. The status may indicate whether the distalend of the EWC 230 is located at a not-in-target location, at thetarget, or at a location adjacent to healthy tissue, and whethertreatment of the target is complete. For example, the color code may beused in a way that a red color indicates that the EWC 230 is at anot-in-target location, a green color indicates that the EWC 230 is at atarget, a yellow color indicates that the EWC 230 is adjacent to healthytissue, and an orange color indicates that the treatment is complete.However, this is an example and is not meant to limit the scope of thisdisclosure. Other status indication systems may be employed as people inthe ordinary skill in the art would apprehend.

Though not described in detail above, with respect to FIG. 1, thenetwork interface 128 enables other computing devices 120, thebronchoscope 115, and the catheter guide assembly 110 to communicatethrough a wired and/or wireless network connection. In FIG. 1, thebronchoscope 115 and catheter guide assembly 110 may transmit or receivemedical images, medical data, and control data to and from the computingdevice 120 via a wired connection. In a case where the network interface128 connects to other computing devices or the bronchoscope 115 andcatheter guide assembly 110 wirelessly, the network interface 128 uses afrequency for communication, which may be different from the frequencythe bronchoscope 115 or the catheter guide assembly 110 uses fortransmitting the captured images.

The memory 126 of computing device 120 may include one or more amongsolid-state storage devices, flash memory chips, mass storage, tapedrive, or any computer-readable storage medium which is connected to aprocessor through a storage controller and a communications bus.Computer readable storage media include non-transitory, volatile,non-volatile, removable, and non-removable media implemented in anymethod or technology for storage of information such ascomputer-readable instructions, data structures, program modules orother data. For example, computer-readable storage media includes randomaccess memory (RAM), read-only memory (ROM), erasable programmable readonly memory (EPROM), electrically erasable programmable read only memory(EEPROM), flash memory or other solid state memory technology, CD-ROM,DVD or other optical storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any othermedium which can be used to store desired information and which can beaccessed by the computing device 120.

In embodiments, the display 122 may work as an input device such thatthe display 122 may receive multiple finger actions, such as pinching orspreading fingers. For example, when fingers are pinched, the portion ofthe displayed image, where the fingers are located on the display 122before pinching, may be zoomed out and, when fingers are spread, theportion of the lung, where the fingers are located on the display 122before spreading, is zoomed in. Or when multiple fingers swipe thedisplay 122 together in one direction, the displayed image may berotated in the same direction as the swiping direction and the amount ofrotation is proportional to a distance and/or a speed of the swipingmotion. These features may be also implemented using the input device129.

The input device 129 is used for inputting data or control information,such as setting values, or text information. The input device 129includes a keyboard, mouse, scanning devices, or other data inputdevices. The input device 129 may be further used to manipulatedisplayed images or the 3D model to zoom in and out, and rotate in anydirection.

The monitoring device 130 is operatively connected with the bronchoscope115 and the computing device 120. The monitoring device 130 includesbuttons and switches for setting settable items of the monitoring device130. The monitoring device 130 may be touch-sensitive and/orvoice-activated, enabling the monitoring device 130 to serve as both aninput and output device. Thus, settable items of the monitoring device130 may be set, changed, or adjusted by using the buttons, touches tothe screen of the monitoring device 130, or voices.

When the bronchoscope 115 captures images of the luminal network of thelung and the captured images do not need to be processed forvisualization for human eyes, the monitoring device 130 may receive anddisplay the captured images on the monitoring device 130 so that aclinician may confirm that the location of the catheter guide assembly110 is in an intended place, particularly for use in confirmation ofregistration.

Although embodiments have been described in detail with reference to theaccompanying drawings for the purpose of illustration and description,it is to be understood that the inventive processes and apparatus arenot to be construed as limited. It will be apparent to those of ordinaryskill in the art that various modifications to the foregoing embodimentsmay be made without departing from the scope of the disclosure.

What is claimed is:
 1. A system for ultrasound (US) interrogationcomprising: a memory storing a three dimensional (3D) model of a luminalnetwork and a pathway plan for navigating a luminal network; anelectromagnetic (EM) board configured to generate an EM field; anextended working channel (EWC) configured to navigate the luminalnetwork toward a target in accordance with a pathway plan; an EM sensorextending distally from a distal end of the EWC and configured to sensethe EM field; a US transducer configured to generate US waves andreceive US waves reflected from the luminal network; and a processorconfigured to process the sensed EM field to synchronize a location ofthe EM sensor in the 3D model, to process the reflected US waves togenerate US images, or to integrate the generated images with the 3Dmodel.
 2. The system according to claim 1, further comprising a displaydevice configured to display the integrated 3D model and US images. 3.The system according to claim 2, wherein the display is furtherconfigured to display a status based on the location of the EM sensor.4. The system according to claim 3, wherein the status indicates whetherthe EM sensor is located at a not-in-target location, the target, or alocation adjacent to healthy tissue.
 5. The system according to claim 3,wherein the status indicates whether treatment of the target iscomplete.
 6. The system according to claim 1, wherein a resolution ofthe generated images is finer than a resolution of the 3D model.
 7. Thesystem according to claim 1, wherein the EM sensor is located at oraround a distal end of the EWC.
 8. The system according to claim 1,further comprising a plurality of reference sensors located on a patientand configured to recognize a breathing pattern of the patient.
 9. Thesystem according to claim 8, further comprising a tracking devicecoupled to the plurality of reference sensors and the EM sensor, andconfigured to identify the location of the EM sensor by compensating forpatient's breathing based on the breathing pattern.
 10. The systemaccording to claim 1, wherein a location of integration of the generatedimages is based on the location of the EM sensor in the 3D model. 11.The system according to claim 1, wherein the processor is furtherconfigured to identify tissue density based on the reflected US waves.12. The system according to claim 1, wherein the processor is furtherconfigured to determine whether a treatment device is at a center of thetarget.
 13. The system according to claim 1, wherein the processor isfurther configured to determine a sufficiency of treatment based on adensity of the target according to the reflected US waves.
 14. Thesystem according to claim 1, wherein the processor is further configuredto detect a size of the target.
 15. The system according to claim 1,wherein the processor is further configured to determine shrinkage ofthe target real-time during and after a treatment of the target.
 16. Thesystem according to claim 1, wherein the generated images show tissueoutside of the luminal network.
 17. The system according to claim 1,wherein the US transducer is inserted inside of the EWC.
 18. The systemaccording to claim 1, wherein the processor is further configured todetermine an offset between the EM sensor and the US transducer.
 19. Thesystem according to claim 18, wherein integration of the generatedimages with the 3D model is based on the offset.
 20. The systemaccording to claim 1, wherein the US transducer is positioned in aforward looking manner before the EM sensor.